-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade Azure Blob Storage SDK to v12 #2573
base: master
Are you sure you want to change the base?
Conversation
There's an admin test that is failing in CI that isn't failing locally, so I'm going to put some debugging in in the next commits to try to figure out what's happening. |
FYI, it is failing for me locally too 👀 Not that it helps much ... but I can possibly investigate a bit myself and see if I find anything. |
If it helps:
|
Thanks! Might be a statefulness thing, I'll start a fresh env. |
...Wouldn't it be better to include actual information in that The check would work the same, and the error could be logged properly before redirect (...as it probably should be, anyway.) |
@samuelhwilliams It was an error due to the tests using an older Azure storage emulator, I've updated them to the official Microsoft hosted emulator now, and they're passing fine. |
@@ -0,0 +1,14 @@ | |||
// For format details, see https://aka.ms/devcontainer.json. | |||
{ | |||
"name": "flask-admin (Python + Azurite)", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the future, I could add Postgres and Mongo to this dev container too, to have a single container that can run all the tests. It should be fairly easy given tests.yaml has the services setup, just copying that to docker-compose.yaml.
from azure.storage.blob import BlobServiceClient | ||
from azure.storage.blob import generate_blob_sas | ||
except ImportError as e: | ||
raise Exception( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the handling of the importerror to make mypy happier. I looked at other optional modules and they all seemed to handle importerrors a bit differently. I found one that did it this way. This seems fine since you should import the module unless you're using it.
Hm, it appears I still don't have an approach for handling unimportable Azure modules that makes mypy happy. Let me know if you have any thoughts about the best practice for importing extra modules in tests (and skipping tests if they don't exist). I'll take another look Monday otherwise. |
I don't think we need to support tests running without azure installed - I'd probably be fine with you removing the try/except around that test import. |
Okay, I've made it so that the tests assume you've got azure-blob-storage installed. I've also made the tests devcontainer bring in Postgres and Mongo too, so I was able to get all the tests passing in my dev container environment, without additional setup. |
self._container_name = container_name | ||
self._connection_string = connection_string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of the changes I made on the s3 admin side when bringing it up to date was to have __init__
take a client instance rather than parameters that get passed the client.
Do you think we should do something similar here and accept an instance of BlobServiceClient
, or is it still fine to just use the connection string?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I think thats nice, as I personally don't typically use connection strings (this was my first time using from_connection_string), so that gives developers more flexibility as to how they connect. I can make that change. That'd be breaking, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be, but this is scheduled to go out for the v2 release where we're making a bunch of breaking changes, so I'm ok with it.
If you'd be happy to, feel free :) 🙏
I'm not sure the |
I also think I'm getting one error in the devcontainer that feels odd: But clearly it passes in CI, so... |
@samuelhwilliams I put the hstore setup in the postAttachCommand, but that doesnt' run until really late in the process, so it's possible you ran the tests before that had been run. I can make it more robust by instead bringing in a Dockerfile for the postgis and putting the CREATE EXTENSIOn in there. @samuelhwilliams Do you get that error when there are no changes locally? Sometimes errors happen with partially aborted tests due to dangling changes to dummy files. |
Apologies - both of these problems were on my end, my checkout of your latest changes failed so I still had an old version of the PR cached. After resetting that, unfortunately I get a new error that looks unrelated - not able to install But I think, yeah, neither of these things are strictly this PR's problem... |
@samuelhwilliams I ran into both install issues. I've been commenting out numpy and manually installing Flask==3.0.0. I wasn't sure whether to tackle those in the PR or not. |
@samuelhwilliams I'll check on the download error, that sounds like a flow I didn't do. (And perhaps isnt covered by tests, given they're passing). And add a link. |
Ah ok that's reassuring for me at least. I think it's fine to leave it for a separate PR; I'll raise an issue on it...
Brill, thanks! |
The error with download_file looks like this Azurite issue: |
…stead of SAS, add prod config to example
Update:
|
FYI: there are no type annotations for AzureFileAdmin and S3FileAdmin currently. When we add them in, we get a mypy error about "multiple values for keyword storage". I think it's confused by the args/kwargs around it. So I have left off type annotations on AzureFileAdmin for now, though I added them to the storage class in a few places. |
Just testing and still seeing errors on download that you said are related to Azure/Azurite#656, and also managed to upload a JPG and get an error from azure when trying to rename it. 🤔 If this is still WIP can we flip it to draft, as I'm not sure whether you want me to re-review it yet 👀 (or can you @ me when it's ready so that I know I need to look 🙏) |
Are you getting those with the local Azure connection or prod? I thought I
tested that rename flow for both local/prod but I can go through it again
and add a pytest so that CI should catch these issues.
…On Sun, Dec 1, 2024 at 8:37 AM Samuel Williams ***@***.***> wrote:
Just testing and still seeing errors on download that you said are related
to Azure/Azurite#656 <Azure/Azurite#656>, and
also managed to upload a JPG and get an error from azure when trying to
rename it. 🤔
If this is still WIP can we flip it to draft, as I'm not sure whether you
want me to re-review it yet 👀
—
Reply to this email directly, view it on GitHub
<#2573 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACIQUT4WYU2P2ZFZPEDRJT2DM3MTAVCNFSM6AAAAABSH4QMS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZHE3DSNRXGU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Using (I don't have a real azure account so might struggle to test there - let me know if I really need to get one 👀) |
Hm, I haven't been able to replicate yet, on this branch in GitHub Codespaces with the dev container configuration. One thing I could do early next week is to ask my colleagues to do a bug bash on this branch, with both local and prod, to see if they encounter any issues (since we all have Azure accounts). That'd be a fun excuse to intro them to flask-admin anyway. Here's my successful rename: And my logs - which aren't super useful except that they show which URL is being requested. From what I've read, Azurite "start_copy_from_url" should work as long as the server is the same (http://127.0.0.1:10000 in this case).
|
I'm bug bash'ing this branch with some colleagues tomorrow, so hopefully we'll replicate that issue. |
if not blob.properties or not blob.properties.has_key("content_settings"): | ||
raise ValueError("Blob has no properties") | ||
mime_type = blob.properties["content_settings"]["content_type"] | ||
blob_file = io.BytesIO() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the blob_file
should be closed here. It will delete the buffer. flask.send_file
doesn't seem to close it. It will be closed when the variable is GC'ed but if you're getting lots of files it could consume a lot of memory
We just did a little bug bash. Possible issues:
Issues that are probably Codespace-related and shouldn't happen outside of Codespaces:
Issues outside of this PR's scope:
I'll move the PR to WIP while I triage those issues. We did not replicate the rename-file issue with Azurite, all attendees were able to rename. |
I tried a larger file, no luck, but can you attach your large file and also tell me what you tried renaming it to? In theory, I should be able to replicate, given we're in a containerized environment. If I still fail, then I'll DM next week on Discord and see if we can hop on a screen share or some thing. |
Fixes #2566
This PR upgrades the azure-storage-blob SDK from v2 to v12, which involved a lot of interface changes. I followed the migration guide @ https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/migration_guide.md and was able to get all the previous functionality working, at least according to the tests.
For ease of testing, I added a simple example app and a devcontainer.azure.json which brings in the Azurite local emulator. That means you can open this repo inside a Codespace or Dev Container with that configuration, and Azure Blob Storage will be running for you.